Data pipeline and deep learning system for autonomous driving

ABSTRACT

An image captured using a sensor on a vehicle is received and decomposed into a plurality of component images. Each component image of the plurality of component images is provided as a different input to a different layer of a plurality of layers of an artificial neural network to determine a result. The result of the artificial neural network is used to at least in part autonomously operate the vehicle.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of, and claims priority to, U.S.patent application Ser. No. 16/013,817 titled “DATA PIPELINE AND DEEPLEARNING SYSTEM FOR AUTONOMOUS DRIVING” and filed on Jun. 20, 2018, thedisclosure of which is hereby incorporated herein by reference in itsentirety.

BACKGROUND OF THE INVENTION

Deep learning systems used to implement autonomous driving typicallyrely on captured sensor data as input. In traditional learning systems,the captured sensor data is made compatible with a deep learning systemby converting the captured data from a sensor format to a formatcompatible with the initial input layer of the learning system. Thisconversion may include compression and down-sampling that can reduce thesignal fidelity of the original sensor data. Moreover, changing sensorsmay require a new conversion process. Therefore, there exists a need fora customized data pipeline that can maximize the signal information fromthe captured sensor data and provide a higher level of signalinformation to the deep learning network for deep learning analysis.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the invention are disclosed in the followingdetailed description and the accompanying drawings.

FIG. 1 is a flow diagram illustrating an embodiment of a process forperforming machine learning processing using a deep learning pipeline.

FIG. 2 is a flow diagram illustrating an embodiment of a process forperforming machine learning processing using a deep learning pipeline.

FIG. 3 is a flow diagram illustrating an embodiment of a process forperforming machine learning processing using component data.

FIG. 4 is a flow diagram illustrating an embodiment of a process forperforming machine learning processing using high-pass and low-passcomponent data.

FIG. 5 is a flow diagram illustrating an embodiment of a process forperforming machine learning processing using high-pass, band-pass, andlow-pass component data.

FIG. 6 is a block diagram illustrating an embodiment of a deep learningsystem for autonomous driving.

DETAILED DESCRIPTION

The invention can be implemented in numerous ways, including as aprocess; an apparatus; a system; a composition of matter; a computerprogram product embodied on a computer readable storage medium; and/or aprocessor, such as a processor configured to execute instructions storedon and/or provided by a memory coupled to the processor. In thisspecification, these implementations, or any other form that theinvention may take, may be referred to as techniques. In general, theorder of the steps of disclosed processes may be altered within thescope of the invention. Unless stated otherwise, a component such as aprocessor or a memory described as being configured to perform a taskmay be implemented as a general component that is temporarily configuredto perform the task at a given time or a specific component that ismanufactured to perform the task. As used herein, the term ‘processor’refers to one or more devices, circuits, and/or processing coresconfigured to process data, such as computer program instructions.

A detailed description of one or more embodiments of the invention isprovided below along with accompanying figures that illustrate theprinciples of the invention. The invention is described in connectionwith such embodiments, but the invention is not limited to anyembodiment. The scope of the invention is limited only by the claims andthe invention encompasses numerous alternatives, modifications andequivalents. Numerous specific details are set forth in the followingdescription in order to provide a thorough understanding of theinvention. These details are provided for the purpose of example and theinvention may be practiced according to the claims without some or allof these specific details. For the purpose of clarity, technicalmaterial that is known in the technical fields related to the inventionhas not been described in detail so that the invention is notunnecessarily obscured.

A data pipeline that extracts and provides sensor data as separatecomponents to a deep learning network for autonomous driving isdisclosed. In some embodiments, autonomous driving is implemented usinga deep learning network and input data received from sensors. Forexample, sensors affixed to a vehicle provide real-time sensor data,such as vision, radar, and ultrasonic data, of the vehicle's surroundingenvironment to a neural network for determining vehicle controlresponses. In some embodiments, the network is implemented usingmultiple layers. The sensor data is extracted into two or more differentdata components based on the signal information of the data. Forexample, feature and/or edge data may be extracted separate from globaldata such as global illumination data into different data components.The different data components retain the targeted relevant data, forexample, data that will eventually be used to identify edges and otherfeatures by a deep learning network. In some embodiments, the differentdata components function as containers that store data highly relevantfor identifying certain targeted features but do not themselves identifyor detect the features. The different data components extract data toensure accurate feature detection at an appropriate stage of a machinelearning network. In some embodiments, the different data components maythen be pre-processed to enhance the particular signal information thatthey contain. The data components can be compressed and/or down-sampledto increase resource and computational efficiency. The different datacomponents are then provided to the deep learning system at differentlayers of the system. The deep learning network is able to accuratelyidentify and detect the features associated with the targeted data(e.g., edges, objects, etc.) of the data component using the signalinformation retained during extraction as input. For example, featureand edge data is provided to the first layer of the network and globaldata to a later layer of the network. By extracting different datacomponents that each retain their respective targeted signalinformation, the network more efficiently processes the sensor data.Instead of receiving the sensor data as an initial input to the network,the network is provided with the most useful information at the mostappropriate layer of the network. In some embodiments, a more completeversion of the captured sensor data is analyzed by the network since thedifferent data components can fully utilize the image resolution oftheir respective components for their intended purposes. For example,input for features and edges can utilize the entire resolution, bitrange, and bit depth for feature and edge data whereas input for globalillumination can utilize the entire resolution, bit range, and bit depthfor global illumination data.

In some embodiments, an image captured using a sensor on a vehicle isreceived. For example, an image is captured from a high dynamic rangeforward-facing camera. As another example, ultrasonic data is capturedfrom a side-facing ultrasonic sensor. In some embodiments, the receivedimage is decomposed into a plurality of component images. For example,feature data is extracted from a captured high dynamic range image. Asanother example, global illumination data is extracted from the capturedhigh dynamic range image. As another example, the image may bedecomposed using high-pass, low-pass, and/or band-pass filters. In someembodiments, each component image of the plurality of component imagesis provided as a different input to a different layer of a plurality oflayers of an artificial neural network to determine a result. Forexample, an artificial neural network such as a convolutional neuralnetwork includes multiple layers for processing input data. Thedifferent component images decomposed from the captured image arepresented as input to different layers of the neural network. Forexample, feature data is presented as input to the first layer of thenetwork and global data is presented as input to a later layer (e.g.,the third layer) of the network. In some embodiments, the result of theartificial neural network is used to at least in part autonomouslyoperate the vehicle. For example, the result of deep learning analysisusing the artificial neural network is used to control the steering,breaking, lighting, and/or warning systems of the vehicle. In someembodiments, the result is used to autonomously match the vehicle'sspeed to traffic conditions, steer the vehicle to follow a navigationalpath, avoid collisions when an object is detected, summon the vehicle toa desired location, and warn the user of potential collisions, amongother autonomous driving applications.

In some embodiments, a vehicle is affixed with multiple sensors forcapturing data. For example, in some embodiments, eight surround camerasare affixed to a vehicle and provide 360 degrees of visibility aroundthe vehicle with a range of up to 250 meters. In some embodiments,camera sensors include a wide forward camera, a narrow forward camera, arear view camera, forward looking side cameras, and/or rearward lookingside cameras. In some embodiments, ultrasonic and radar sensors are usedto capture surrounding details. For example, twelve ultrasonic sensorsmay be affixed to the vehicle to detect both hard and soft objects. Insome embodiments, a forward-facing radar is utilized to capture data ofthe surrounding environment. In various embodiments, radar sensors areable to capture surrounding detail despite heavy rain, fog, dust, andother vehicles. The various sensors are used to capture the environmentsurrounding the vehicle and the captured image is provided for deeplearning analysis.

Using data captured from sensors and analyzed using the disclosed deeplearning system, a machine learning result is determined for autonomousdriving. In various embodiments, the machine learning result is providedto a vehicle control module for implementing autonomous drivingfeatures. For example, a vehicle control module can be used to controlthe steering, braking, warning systems, and/or lighting of the vehicle.In some embodiments, the vehicle is controlled to navigate roads, matchthe speed of the vehicle to traffic conditions, keep the vehicle withina lane, automatically change lanes without requiring driver input,transition the vehicle from one freeway to another, exit the freewaywhen approaching a destination, self-park the vehicle, and summon thevehicle to and from a parking spot, among other autonomous drivingapplications. In some embodiments, the autonomous driving featuresinclude identifying opportunities to move the vehicle into a faster lanewhen behind slower traffic. In some embodiments, the machine learningresult is used determine when autonomous driving without driverinteraction is appropriate and when it should be disabled. In variousembodiments, the machine learning result is used to assist a driver indriving the vehicle.

In some embodiments, the machine learning result is used to implement aself-parking mode where the vehicle will automatically search for aparking spot and park the vehicle. In some embodiments, the machinelearning result is used to navigate the vehicle using a destination froma user's calendar. In various embodiments, the machine learning resultis used to implement autonomous driving safety features such ascollision avoidance and automatic emergency braking. For example, insome embodiments, the deep learning system detects objects that mayimpact with the vehicle and the vehicle control module applies thebrakes accordingly. In some embodiments, the vehicle control module usesthe deep learning analysis to implement a side, front, and/or rearcollision warning that warns the user of the vehicle of potentialcollisions with obstacles alongside, in front, or behind the vehicle. Invarious embodiments, the vehicle control module can activate warningsystems such as collision alerts, audio alerts, visual alerts, and/orphysical alerts (such as vibration alerts), among others, to inform theuser of an emergency situation or when the driver's attention isnecessary. In some embodiments, the vehicle control module can initiatea communication response such as an emergency response call, a textmessage, a network update, and/or another communication response asappropriate, for example, to inform another party of an emergencysituation. In some embodiments, the vehicle control module can adjustthe lighting including the high/low beams, brake lights, interior light,emergency lights, etc. based on the deep learning analysis results. Insome embodiments, the vehicle control module can further adjust theaudio in or around the vehicle including using the horn, modifying theaudio (e.g., music, phone calls, etc.) playing from the vehicle's soundsystem, adjusting the volume of the sound system, playing audio alerts,enabling a microphone, etc. based on deep learning analysis results.

FIG. 1 is a flow diagram illustrating an embodiment of a process forperforming machine learning processing using a deep learning pipeline.For example, the process of FIG. 1 may be utilized to implementautonomous driving features for self-driving and driver-assistedautomobiles to improve safety and to reduce the risk of accidents. Insome embodiments, the process of FIG. 1 pre-processes data captured bysensors for deep learning analysis. By pre-processing the sensor data,the data provided for deep learning analysis is enhanced and results ina more accurate result for controlling a vehicle. In some embodiments,the pre-processing addresses data mismatches between data captured bysensors and data expected by a neural network for deep learning.

At 101, sensor data is received. For example, sensor data is captured byone or more sensors affixed to a vehicle. In some embodiments, thesensors are affixed to the environment and/or other vehicles and data isreceived remotely. In various embodiments, the sensor data is imagedata, such as RGB or YUV channels of an image. In some embodiments, thesensor data is captured using a high dynamic range camera. In someembodiments, the sensor data is radar, LiDAR, and/or ultrasonic data. Invarious embodiments, LiDAR data is data captured using laser light andmay includes techniques referred to as Light Detection And Ranging aswell as Laser Imaging, Detection and Ranging. In various embodiments,the bit depth of the sensor data exceeds the bit depth of the neuralnetwork for deep learning analysis.

At 103, data pre-processing is performed on the sensor data. In someembodiments, one or more pre-processing passes may be performed on thesensor data. For example, the data may be first pre-processed to removenoise, to correct for alignment issues and/or blurring, etc. In someembodiments, two or more different filtering passes are performed on thedata. For example, a high-pass filter may be performed on the data and alow-pass filter may be performed on the data. In some embodiments, oneor more band pass filters may be performed. For example, one or moreband passes may be performed on the data in addition to a high-pass anda low-pass. In various embodiments, the sensor data is separated intotwo or more data sets such as a high-pass data set and a low-pass dataset. In some embodiments, one or more band pass data sets are alsocreated. In various embodiments, the different data sets are differentcomponents of the sensor data.

In some embodiments, the different components created by pre-processingthe data include a feature and/or edge component and a global datacomponent. In various embodiments, the feature and/or edge component iscreated by performing a high-pass or band-pass filter on the sensor dataand the global data component is created by performing a low-pass orband-pass filter on the sensor data. In some embodiments, one or moredifferent filter techniques may be used to extract feature/edge dataand/or global data.

In various embodiments, one or more components of the sensor data areprocessed. For example, a high-pass component may be processed byremoving noise from and/or enhancing local contrast for the image data.In some embodiments, the low-pass component is compressed and/ordown-sampled. In various embodiments, different components arecompressed and/or down-sampled. For example, components may becompressed, resized, and/or down-sampled as appropriate to adjust thesize and/or resolution of the data for inputting the data to a layer ofa machine learning model. In some embodiments, the bit depth of thesensor data is adjusted. For example, a data channel of a cameracapturing data at 20-bits or another appropriate bit depth is compressedor quantized to 8-bits to prepare the channel for an 8-bit machinelearning model. In some embodiments, one or more sensors capture data ata bit depth of 12-bits, 16-bits, 20-bits, 32-bits, or anotherappropriate bit depth that is larger than the bit depth used by the deeplearning network.

In various embodiments, the pre-processing performed at 103 is performedby an image pre-processor. In some embodiments, the image pre-processoris a graphics processing unit (GPU), a central processing unit (CPU), anartificial intelligence (AI) processor, an image signal processor, atone-mapper processor, or other similar hardware processor. In variousembodiments, different image pre-processors are used to extract and/orpre-process different data components in parallel.

At 105, deep learning analysis is performed. For example, deep learninganalysis is performed using a machine learning model such as anartificial neural network. In various embodiments, the deep learninganalysis receives the processed sensor data for 103 as input. In someembodiments, the processed sensor data is received at 105 as multipledifferent components, such as a high-pass data component and a low-passdata component. In some embodiments, the different data components arereceived as inputs to different layers of the machine learning model.For example, a neural network receives a high-pass component as aninitial input to the first layer of the network and a low-pass componentas input to a subsequent layer of the network.

At 107, the results of the deep learning analysis are provided forvehicle control. For example, the results may be provided to a vehiclecontrol module to adjust the speed and/or steering of the vehicle. Invarious embodiments, the results are provided to implement autonomousdriving functionality. For example, the results may indicate an objectthat should be avoided by steering the vehicle. As another example, theresults may indicate a merging car that should be avoided by braking andchanging the vehicle's positioning in the lane.

FIG. 2 is a flow diagram illustrating an embodiment of a process forperforming machine learning processing using a deep learning pipeline.For example, the process of FIG. 2 may be utilized to pre-process sensordata, extract image components from the sensor data, pre-process theextracted image components, and then provide the components for deeplearning analysis. The results of deep learning analysis may be used toimplement autonomous driving to improve safety and to reduce the risk ofaccidents. In some embodiments, the process of FIG. 2 is used to performthe process of FIG. 1 . In some embodiments, step 201 is performed at101 of FIG. 1 ; steps 203, 205, 207, and/or 209 are performed at 103 ofFIG. 1 ; and/or step 211 is performed at 105 and/or 107 of FIG. 1 . Byprocessing the extracted components of the sensor data, the processeddata provided to a machine learning model is enhanced to achievesuperior results from deep learning analysis rather than using otherwisenon-enhanced data. In the example shown, the results of the deeplearning analysis are used for vehicle control.

At 201, sensor data is received. In various embodiments, the sensor datais image data captured from a sensor such as a high dynamic rangecamera. In some embodiments, the sensor data is captured from one ormore different sensors. In some embodiments, the image data is capturedusing a 12-bit or higher bit depth to increase the fidelity of the data.

At 203, the data is pre-processed. In some embodiments, the data ispre-processed using an image pre-processor such as an image signalprocessor, a graphics processing unit (GPU), a tone-mapper processor, acentral processing unit (CPU), an artificial intelligence (AI)processor, or other similar hardware processor. In various embodiments,linearization, demosaicing, and/or another processing techniques may beperformed on the captured sensor data. In various embodiments,pre-processing is performed on the high-resolution sensor data toenhance the fidelity of the captured data and/or to reduce theintroduction of errors by subsequent steps. In some embodiments, thepre-processing step is optional.

At 205, one or more image components are extracted. In some embodiments,two image components are extracted. For example, a feature/edge datacomponent of the sensor data is extracted and a global data component ofthe sensor data is extracted. In some embodiments, a high-pass componentand a low-pass component of the sensor data are extracted. In someembodiments, one or more additional band-pass components are extractedfrom the sensor data. In various embodiments, high-pass, low-pass,and/or band-pass filters are used to extract different components of thesensor data. In some embodiments, the image components are extractedusing a tone mapper. In some embodiments, the global data and/orlow-pass component data is extracted by down-sampling the sensor datausing a binning or similar technique. In various embodiments, theextraction retains and saves the targeted signal information as an imagedata component but does not actually detect or identify the featuresrelated to the targeted information. For example, the extraction of animage component corresponding to edge data results in an image componentwith targeted signal information for accurately identifying edges butthe extraction performed at 205 does not detect the existence of edgesin the sensor data.

In some embodiments, the image data component extracted for a firstlayer of a machine learning network is extracted using a process thatpreserves the response of the first layer of the deep learning analysis.For example, the relevant signal information for the first layer ispreserved such that the result of the analysis performed on the imagecomponent after the analysis of the first layer is similar to theanalysis performed on the corresponding sensor data prior to extractioninto image components. In various embodiments, the results are preservedfor filters as small as a 5×5 matrix filter.

In some embodiments, an extracted data component is created by combiningmultiple channels of the captured image into one or more channels. Forexample, red, green, and blue channels may be averaged to create a newchannel for a data component. In various embodiments, an extracted datacomponent may be constructed from one or more different channels of thesource capture data and/or one or more different captured images ofdifferent sensors. For example, data from multiple sensors may becombined into a single data component.

In some embodiments, an image pre-processor such as the pre-processor ofstep 203 is used to extract the different components. In someembodiments, an image signal processor may be used to extract thedifferent components. In various embodiments, a graphics processing unit(GPU) may be used to extract the different components. In someembodiments, a different pre-processor is used to extract differentcomponents so that multiple components can be extracted in parallel. Forexample, an image signal processor may be used to extract a high-passcomponent and a GPU may be used to extract a low-pass component. Asanother example, an image signal processor may be used to extract alow-pass component and a GPU may be used to extract a high-passcomponent. In some embodiments, a tone-mapper processor is used toextract an image component (such as a high-pass component) and a GPU isused to extract a separate image component (such as a low-passcomponent) in parallel. In some embodiments, the tone-mapper is part ofan image signal processor. In some embodiments, multiple instances ofsimilar pre-processors exist to perform extractions in parallel.

At 207, component pre-processing is performed. In some embodiments, animage pre-processor such as the pre-processor of step 203 and/or 205 isused to pre-process one or more components. In some embodiments, adifferent pre-processor is used to pre-process different components sothat the pre-processing can be performed on the different components inparallel. For example, an image signal processor may be used to processa high-pass component and a graphics processing unit (GPU) may be usedto process a low-pass component. In some embodiments, a tone-mapperprocessor is used to process one image component and a GPU is used toprocess a separate image component in parallel. In some embodiments,multiple instances of similar pre-processors exist for processingdifferent components in parallel.

In various embodiments, the pre-processing includes down-sampling and/orcompressing the image component data. In some embodiments, thepre-processing includes removing noise from the component data. In someembodiments, the pre-processing includes compressing or quantizing thecaptured data from 20-bit down to 8-bit data fields. In someembodiments, the pre-processing includes converting the size of theimage component to a lower resolution. For example, an image componentmay be half, a quarter, an eighth, a sixteenth, one thirty-second, onesixty-fourth, or another appropriate scaling of the original sensorimage size. In various embodiments, an image component is reduced to asize appropriate to the input layer of the machine learning model.

At 209, components are provided to the appropriate network layer of thedeep learning network. For example, different components may be providedto different layers of the network. In some embodiments, the network isa neural network with multiple layers. For example, the first layer of aneural network receives as input high-pass component data. One of thesubsequent network layers receives as input low-pass component datacorresponding to global illumination data. In various embodiments, thedifferent components extracted at 205 and pre-processed at 207 arereceived at different layers of the neural network. As another example,a feature and/or edge data component is provided as input to the firstlayer of a deep learning network such as an artificial neural network. Aglobal data component is provided to a subsequent layer and can beprovided as a compressed and/or down-sampled version of the data sincethe global data does not require as much precision as feature and/oredge component data. In various embodiments, global data is more easilycompressed without losing information and can be provided at a laterlayer of the network.

In some embodiments, the machine learning model is made up of multiplesequential layers where the one or more subsequent layers receive inputdata that has a size property that is smaller in size than a previouslayer. For example, the first layer to a network may receive an imagesize similar to the capture image size. Subsequent layers may receiveinput data that is a half or a quarter of the capture image size. Thereduction in input data size reduces the computation of subsequentlayers and improves the efficiency of the deep learning analysis. Byproviding the sensor input data as different components and at differentlayers, computational efficiency is increased. Earlier layers of thenetwork require increased computation in particular because the amountof data and the data size is larger than subsequent layers. Subsequentlayers may be more efficient to compute since the input data has beencompressed by previous layers of the network and/or the pre-processingat 207.

At 211, results of the deep learning analysis are provided for vehiclecontrol. For example, machine learning results using the processed imagecomponents may be utilized to control a vehicle's movement. In someembodiments, the results correspond to vehicle control actions. Forexample, results may correspond to the speed and steering of thevehicle. In some embodiments, the results are received by a vehiclecontrol module used to help maneuver the vehicle. In some embodiments,the results are utilized to improve the safety of the vehicle. Invarious embodiments, the results provided at 211 are determined byperforming a deep learning analysis on the components provided at 209.

FIG. 3 is a flow diagram illustrating an embodiment of a process forperforming machine learning processing using component data. In theexample shown, the process of FIG. 3 is used to extract feature and edgedata from sensor data separate from global data. The two data sets arethen fed into a deep learning network at different stages to infervehicle control results. By separating the two components and providingthem at different stages, the initial layers of the network can dedicatecomputational resources to initial edges and feature detection. In someembodiments, the initial stages dedicate resources to the initialidentification of the objects such as roads, lane markers, obstacles,vehicles, pedestrians, traffic signs, etc. Subsequent layers can utilizethe global data in a more computational efficient manner since theglobal data is less resource intensive. Since machine learning can becomputational and data intensive, a data pipeline utilizing differentimage components at different stages is utilized to increase theefficiency of the deep learning computation and to reduce data resourcerequirements needed for the analysis. In some embodiments, the processof FIG. 3 is used to perform the process of FIG. 1 and/or FIG. 2 . Insome embodiments, step 301 is performed at 101 of FIG. 1 and/or at 201of FIG. 2 ; step 303 is performed at 103 of FIG. 1 and/or at 203 of FIG.2 ; steps 311 and/or 321 are performed at 103 of FIG. 1 and/or at 205 ofFIG. 2 ; steps 313, 323, and/or 325 are performed at 103 of FIG. 1and/or at 207 and 209 of FIG. 2 ; steps 315 and/or 335 are performed at105 of FIG. 1 and/or at 211 of FIG. 2 ; and/or step 337 is performed at107 of FIG. 1 and/or at 211 of FIG. 2 .

At 301, sensor data is received. In various embodiments, the sensor datais data captured by one or more sensors of the vehicle. In someembodiments, the sensor data is received as described with respect tostep 101 of FIG. 1 and/or step 201 of FIG. 2 .

At 303, data pre-processing is performed. For example, the sensor datais enhanced by pre-processing the data. In some embodiments, the data iscleaned up, for example, by performing a de-noising, alignment, or otherappropriate filter. In various embodiments, the data is pre-processed asdescribed with respect to step 103 of FIG. 1 and/or step 203 of FIG. 2 .In the example shown, processing continues to steps 311 and 321. In someembodiments, processing at 311 and 321 are run in parallel to extractand process different components of the sensor data. In someembodiments, each branch of processing (e.g., the branch starting at 311and the branch starting at 321) is run sequentially or pipelined. Forexample, processing is performed starting with step 311 to prepare datafor the initial layers of a network. In some embodiments, thepre-processing step is optional.

At 311, feature and/or edge data is extracted from the sensor data. Forexample, feature data and/or edge data is extracted from the capturedsensor data into a component data. In some embodiments, the componentdata retains the relevant signal information from the sensor data foridentifying features and/or edges. In various embodiments, theextraction process preserves the signal information critical foridentifying and detecting features and/or edges and does not actuallyidentify or detect the features or edges from the sensor data. Invarious embodiments, the features and/or edges are detected during oneor more analysis steps at 315 and/or 335. In some embodiments, theextracted feature and/or edge data has the same bit depth as theoriginal captured data. In some embodiments, the extracted data isfeature data, edge data, or a combination of feature and edge data. Insome embodiments, a high-pass filter is used to extract feature and/oredge data from the sensor data. In various embodiments, a tone-mapperprocessor is calibrated to extract feature and/or edge data from thesensor data.

At 313, pre-processing is performed on the feature and/or edge data. Forexample, a de-noising filter may be applied to the data to improve thesignal quality. As another example, different pre-processing techniquessuch as local contrast enhancement, gain adjustment, thresholding, noisefiltering, etc. may be applied to enhance the feature and edge dataprior to deep learning analysis. In various embodiments, thepre-processing is customized to enhance the feature and edge propertiesof the data rather than applying a more generic pre-processing techniqueto the sensor data as a whole. In some embodiments, the pre-processingincludes performing a compression and/or down-sampling on the extracteddata. In some embodiments, the pre-processing step at 313 is optional.

At 315, an initial analysis is performed using the feature and/or edgedata. In some embodiments, the initial analysis is a deep learninganalysis using a machine learning model such as a neural network. Invarious embodiments, the initial analysis receives the feature and edgedata as input to the first layer of the network. In some embodiments,the initial layer of the network prioritizes the detection of featuresand/or edges in the captured image. In various embodiments, the deeplearning analysis is performed using an artificial neural network suchas a convolutional neural network. In some embodiments, the analysis isrun on an artificial intelligence (AI) processor.

At 321, global data is extracted from the sensor data. For example,global data is extracted from the captured sensor data into a componentdata. In some embodiments, the global data corresponds to globalillumination data. In some embodiments, the extracted global data hasthe same bit depth as the original captured data. In some embodiments, alow-pass filter is used to extract global data from the sensor data. Invarious embodiments, a tone-mapper processor is calibrated to extractglobal data from the sensor data. Other techniques, such as binning,resampling, and down-sampling may also be used to extract global data.In various embodiments, the extraction process retains data likely to beglobally relevant and does not identify and detect the global featuresfrom the sensor data. In various embodiments, the global features aredetected by the analysis performed at 335.

At 323, pre-processing is performed on the global data. For example, ade-noising filter may be applied to the data to improve the signalquality. As another example, different pre-processing techniques such aslocal contrast enhancement, gain adjustment, thresholding, noisefiltering, etc. may be applied to enhance the global data prior to deeplearning analysis. In various embodiments, the pre-processing iscustomized to enhance the properties of the global data rather thanapplying a more generic pre-processing technique to the sensor data as awhole. In some embodiments, the pre-processing of the global dataincludes compressing the data. In some embodiments, the pre-processingstep at 323 is optional.

At 325, the global data is down-sampled. For example, the resolution ofthe global data is reduced. In some embodiments, the global data isreduced in size to improve the computational efficiency of analyzing thedata and to configure the global data as input to a later layer of thedeep learning network. In some embodiments, the global data isdown-sampled by binning, resampling, or another appropriate technique.In some embodiments, the down-sampling is performed using a graphicalprocessing unit (GPU) or an image signal processor. In variousembodiments, down-sampling is appropriate for global data since theglobal data does not have the same resolution requirements as featureand/or edge data. In some embodiments, the down-sampling performed at325 is performed when the global data is extracted at 321.

At 335, additional deep learning analysis is performed using results ofthe deep learning analysis on the feature and/or edge data and theglobal data as input. In various embodiments, the deep learning analysisreceives as input the global data at a later layer of the deep learningnetwork. In various embodiments, the expected input data size at thelayer receiving the global data is smaller than the expected input datasize of the initial input layer. For example, the input size for theglobal data input layer may be a half or a quarter of the input size forthe initial layer of the deep learning network. In some embodiments, thelater layers of the network utilize global data to enhance the resultsof the initial layers. In various embodiments, the deep learninganalysis is performed and a vehicle control result is determined. Forexample, a vehicle control result is determined using a convolutionalneural network. In some embodiments, the analysis is run on anartificial intelligence (AI) processor.

At 337, the results of deep learning analysis are provided for vehiclecontrol. For example, machine learning results using the extracted andprocessed image components are utilized to control a vehicle's movement.In some embodiments, the results correspond to vehicle control actions.In some embodiments, the results are provided as described with respectto step 107 of FIG. 1 and/or step 211 of FIG. 2 .

FIG. 4 is a flow diagram illustrating an embodiment of a process forperforming machine learning processing using high-pass and low-passcomponent data. In the example shown, the process of FIG. 4 is used toextract two data components from sensor data and to provide thecomponents to different layers of a deep learning network such as anartificial neural network. The two components are extracted using ahigh-pass and low-pass filter. In various embodiments, the results areused to implement autonomous driving with improved precision, safety,and/or comfort results. In some embodiments, the process of FIG. 4 isused to perform the process of FIGS. 1, 2 , and/or 3. In someembodiments, step 401 is performed at 103 of FIG. 1 , at 203 of FIG. 2 ,and/or at 303 of FIG. 3 ; step 403 is performed at 103 of FIG. 1 , at205 of FIG. 2 , and/or at 311 of FIG. 3 ; step 413 is performed at 103of FIG. 1 , at 205 of FIG. 2 , and/or at 321 of FIG. 3 ; step 405 isperformed at 103 of FIG. 1 , at 207 and 209 of FIG. 2 , and/or at 313 ofFIG. 3 ; steps 415 and 417 are performed at 103 of FIG. 1 , at 207 and209 of FIG. 2 , and/or at 323 and 325 of FIG. 3 ; step 407 is performedat 105 of FIG. 1 , at 211 of FIG. 2 , and/or at 315 of FIG. 3 ; and/orstep 421 is performed at 105 of FIG. 1 , at 211 of FIG. 2 , and/or at335 of FIG. 3 .

At 401, data is pre-processed. In some embodiments, the data is thesensor data captured from one or more sensors such as high dynamic rangecamera, radar, ultrasonic, and/or LiDAR sensors. In various embodiments,the data is pre-processed as described with respect to 103 of FIG. 1,203 of FIG. 2 , and/or 303 of FIG. 3 . Once the data is pre-processed,processing continues to 403 and 413. In some embodiments, steps 403 and413 are run in parallel.

At 403, a high-pass filter is performed on the data. For example, ahigh-pass filter is performed on the captured sensor data to extracthigh-pass component data. In some embodiments, the high-pass filter isperformed using a graphics processing unit (GPU), a tone-mapperprocesser, an image signal processor, or other image pre-processor. Insome embodiments, the high-pass data component represents featuresand/or edges of the captured sensor data. In various embodiments, thehigh-pass filter is constructed to preserve the response of the firstlayer of a deep learning process. For example, a high-pass filter isconstructed to preserve the response to a small filter at the top of amachine learning network. The relevant signal information for the firstlayer of the network is preserved such that the result of the analysisperformed on a high-pass component data after the first layer is similarto the analysis performed on non-filtered data after the first layer. Invarious embodiments, the results are preserved for filters as small as a5×5 matrix filter.

At 413, a low-pass filter is performed on the data. For example, alow-pass filter is performed on the captured sensor data to extractlow-pass component data. In some embodiments, the low-pass filter isperformed using a graphics processing unit (GPU), a tone-mapperprocesser, an image signal processor, or other image pre-processor. Insome embodiments, the low-pass data component represents global data ofthe captured sensor data such as global illumination data.

In various embodiments, the filtering performed at 403 and 413 may usethe same or different image pre-processors. For example, a tone-mapperprocessor is used to extract a high-pass data component and a graphicsprocessing unit (GPU) is used to extract a low-pass data component. Insome embodiments, the high-pass or low-pass data is extracted bysubtracting one of the data components from the original captured data.

At 405 and 415, post-processing is performed on the respective high-passand low-pass data components. In various embodiments, differentpost-processing techniques are utilized to enhance the signal qualityand/or to reduce the amount of data required to represent the data. Forexample, a de-noising, demosaicing, local contrast enhancement, gainadjustment, and/or thresholding process, among others, may be performedon the respective high-pass and/or low-pass data components. In someembodiments, the data components are compressed and/or down-sampled. Forexample, once the high-pass and/or low-pass data is extracted, therespective data components may be compressed to more efficiently utilizethe entire bit depth range. In some embodiments, the respective datacomponents are compressed or quantized from a higher bit depth ascaptured by sensors to a lower bit depth compatible with the deeplearning network. For example, a sensor data captured at 12-bits,16-bits, 20-bits, 32-bits, or another appropriate bit depth per channelmay be compressed or quantized to a lower bit depth such as 8-bits perchannel. In some embodiments, the post-processing steps at 405 and/or415 are optional.

At 417, the low-pass data component is down-sampled. In variousembodiments, the low-pass data component is fed into the network at alater stage of the network and may be down-sampled to a more efficientresource size. For example, a low-pass data component may be extractedat the full sensor size and reduced to a half or a quarter of theoriginal size. Other percentages of reductions are possible as well. Invarious embodiments, the low-pass data is down-sampled but retains therelevant signal information. In many scenarios, the low-pass data can beeasily down-sampled without losing signal information. By down-samplingthe data, the data is more easily and quickly analyzed at a later layerin the deep learning network.

At 407, deep learning analysis is performed on the high-pass datacomponent. In some embodiments, the high-pass data component is fed intothe initial layer of the deep learning network and represents the mostsignificant data for feature and edge detection. In various embodiments,the results of the deep learning analysis on the first layer using thehigh-pass data component are fed into subsequent layers of the network.For example, a neural network may include multiple layers, for example,five or more layers. The first layer receives the high-pass datacomponent as input and the second layer receives the results of the deeplearning analysis performed by the first layer. In various embodiments,the second or later layer receives the low-pass data components asadditional input to perform additional deep learning analysis.

At 421, additional deep learning analysis is performed using the resultsof the analysis performed at 407 and the low-pass data componentdown-sampled at 417. In various embodiments, the deep learning analysisinfers a vehicle control result. For example, the result of the deeplearning analysis at 407 and 421 is used to control the vehicle forautonomous driving.

FIG. 5 is a flow diagram illustrating an embodiment of a process forperforming machine learning processing using high-pass, band-pass, andlow-pass component data. In the example shown, the process of FIG. 5 isused to extract three or more data components from sensor data and toprovide the components at different layers of a deep learning networksuch as an artificial neural network. Similar to the process of FIG. 4 ,a high-pass and low-pass component is extracted. In addition, theprocess of FIG. 5 extracts one or more band-pass data components. Invarious embodiments, the decomposition of the sensor data into multiplecomponents that are provided to different layers of the deep learningnetwork allows the deep learning analysis to emphasize different sets ofdata at different layers of the network.

In some embodiments, the process of FIG. 5 is used to perform theprocess of FIGS. 1, 2, 3 , and/or 4. In some embodiments, step 501 isperformed at 103 of FIG. 1 , at 203 of FIG. 2 , at 303 of FIG. 3 ,and/or at 401 of FIG. 4 . In some embodiments, step 503 is performed at103 of FIG. 1 , at 205 of FIG. 2 , at 311 of FIG. 3 , and/or at 403 ofFIG. 4 ; step 513 is performed at 103 of FIG. 1 , at 205 of FIG. 2 ,and/or at 311 or 321 of FIG. 3 ; and/or step 523 is performed at 103 ofFIG. 1 , at 205 of FIG. 2 , at 321 of FIG. 3 , and/or at 413 of FIG. 4 .In some embodiments, step 505 is performed at 103 of FIG. 1 , at 207 and209 of FIG. 2 , at 313 of FIG. 3 , and/or at step 405 of FIG. 4 ; step515 is performed at 103 of FIG. 1 , at 207 and 209 of FIG. 2 , at 313,323 and/or 325 of FIG. 3 , and/or at 405, 415, and/or 417 of FIG. 4 ;and/or step 525 is performed at 103 of FIG. 1 , at 207 and 209 of FIG. 2, at 323 and 325 of FIG. 3 , and/or at 415 and 417 of FIG. 4 . In someembodiments, step 537 is performed at 105 of FIG. 1 , at 211 of FIG. 2 ,at 315 and 335 of FIG. 3 ; and/or at 407 and 421 of FIG. 4 .

At 501, data is pre-processed. In some embodiments, the data is thesensor data captured from one or more sensors such as high dynamic rangecamera, radar, ultrasonic, and/or LiDAR sensors. In various embodiments,the data is pre-processed as described with respect to 103 of FIG. 1,203 of FIG. 2, 303 of FIG. 3 , and/or 401 of FIG. 4 . Once the data ispre-processed, processing continues to 503, 513, and 523. In someembodiments, steps 503, 513, and 523 are run in parallel.

At 503, a high-pass filter is performed on the data. For example, ahigh-pass filter is performed on the captured sensor data to extracthigh-pass component data. In some embodiments, the high-pass filter isperformed using a graphics processing unit (GPU), a tone-mapperprocesser, an image signal processor, or another image pre-processor. Insome embodiments, the high-pass data component represents featuresand/or edges of the captured sensor data.

At 513, one or more band-pass filters are performed on the data toextract one or more band-pass data components. For example, a band-passfilter is performed on the captured sensor data to extract componentdata that includes a mix of feature, edge, intermediate, and/or globaldata. In various embodiments, one more band-pass components may beextracted. In some embodiments, the low-pass filter is performed using agraphics processing unit (GPU), a tone-mapper processer, an image signalprocessor, or another image pre-processor. In some embodiments, theband-pass data component represents data that is neither primarilyedge/feature data nor primarily global data of the captured sensor data.In some embodiments, the band-pass data is utilized to preserve datafidelity that may be lost using only a high-pass data component and alow-pass data component.

At 523, a low-pass filter is performed on the data. For example, alow-pass filter is performed on the captured sensor data to extractlow-pass component data. In some embodiments, the low-pass filter isperformed using a graphics processing unit (GPU), a tone-mapperprocesser, an image signal processor, or other image pre-processor. Insome embodiments, the low-pass data component represents global data ofthe captured sensor data such as global illumination data.

In various embodiments, the filtering performed at 503, 513, and 523 mayuse the same or different image pre-processors. For example, atone-mapper processor is used to extract a high-pass data component anda graphics processing unit (GPU) is used to extract band-pass, and/orlow-pass data components. In some embodiments, data components areextracted by subtracting one or more data components from the originalcaptured data.

At 505, 515, and 525, post-processing is performed on the respectivehigh-pass, band-pass, and low-pass data components. In variousembodiments, different post-processing techniques are utilized toenhance the signal quality and/or to reduce the amount of data requiredto represent the data. In some embodiments, the different components arecompressed and/or down-sampled to the appropriate size for the networklayer receiving the data component. In various embodiments, thehigh-pass data will have a higher resolution than the band-pass data andthe band-pass data will have a higher resolution than the low-pass data.In some embodiments, different band-pass data components will also havedifferent resolutions as appropriate for the network layer each isprovided as input for. In some embodiments, the respective datacomponents are compressed or quantized from a higher bit depth ascaptured by sensors to a lower bit depth compatible with the deeplearning network. For example, a sensor data captured at 12-bits perchannel may be compressed or quantized to 8-bits per channel. In variousembodiments, the pre-processing filters are applied as described withrespect to 207 of FIG. 2 and/or 405, 415 , and/or 417 of FIG. 4 .

At 537, deep learning analysis is performed using the data componentresults of 505, 515, and 525. In some embodiments, the high-pass datacomponent is fed into the initial layer of the deep learning network andrepresents the most significant data for feature and edge detection. Theone or more band-pass data components are fed into middle layer(s) ofthe network and include additional data for identifying features/edgesand/or beneficial intermediate or global information. The low-pass datacomponent is fed into a later layer of the network and includes globalinformation to improve the analysis results of the deep learningnetwork. In performing the deep learning analysis, additional datacomponents representing different sensor data are fed into differentlayers as the analysis progresses to increase the accuracy of theresult. In various embodiments, the deep learning analysis infers avehicle control result. For example, the result of the deep learninganalysis is used to control the vehicle for autonomous driving. In someembodiments, a machine learning result is provided to a vehicle controlmodule to at least in part autonomously operate a vehicle.

FIG. 6 is a block diagram illustrating an embodiment of a deep learningsystem for autonomous driving. In some embodiments, the deep learningsystem of FIG. 6 may be used to implement autonomous driving featuresfor self-driving and driver-assisted automobiles. For example, usingsensors affixed to a vehicle, sensor data is captured, processed asdifferent input components, and fed into different stages of a deeplearning network. The result of deep learning analysis is used by avehicle control module to assist in the operation of the vehicle. Insome embodiments, the vehicle control module is utilized forself-driving or driver-assisted operation of the vehicle. In variousembodiments, the processes of FIGS. 1-5 utilize a deep learning systemsuch as the one described in FIG. 6 .

In the example shown, deep learning system 600 is a deep learningnetwork that includes sensors 601, image pre-processor 603, deeplearning network 605, artificial intelligence (AI) processor 607,vehicle control module 609, and network interface 611. In variousembodiments, the different components are communicatively connected. Forexample, sensor data from sensors 601 is fed to image pre-processor 603.Processed sensor data components of image pre-processor 603 are fed todeep learning network 605 running on AI processor 607. The output ofdeep learning network 605 running on AI processor 607 is fed to vehiclecontrol module 609. In various embodiment, network interface 611 is usedto communicate with remote servers, to make phone calls, to send and/orreceive text messages, etc. based on the autonomous operation of thevehicle.

In some embodiments, sensors 601 include one or more sensors. In variousembodiments, sensors 601 may be affixed to a vehicle, at differentlocations of the vehicle, and/or oriented in one or more differentdirections. For example, sensors 601 may be affixed to the front, sides,rear, and/or roof, etc. of the vehicle in forward-facing, rear-facing,side-facing, etc. directions. In some embodiments, sensors 610 may beimage sensors such as high dynamic range cameras. In some embodiments,sensors 601 include non-visual sensors. In some embodiments, sensors 601include radar, LiDAR, and/or ultrasonic sensors, among others. In someembodiments, sensors 601 are not mounted to the vehicle with vehiclecontrol module 609. For example, sensors 601 may be mounted onneighboring vehicles and/or affixed to the road or environment and areincluded as part of a deep learning system for capturing sensor data.

In some embodiments, image pre-processor 603 is used to pre-processsensor data of sensors 601. For example, image pre-processor 603 may beused to pre-process the sensor data, split sensor data into one or morecomponents, and/or post-process the one or more components. In someembodiments, image pre-processor 603 is a graphics processing unit(GPU), a central processing unit (CPU), an image signal processor, or aspecialized image processor. In various embodiments, image pre-processor603 is a tone-mapper processor to process high dynamic range data. Insome embodiments, image pre-processor 603 is implemented as part ofartificial intelligence (AI) processor 607. For example, imagepre-processor 603 may be a component of AI processor 607.

In some embodiments, deep learning network 605 is a deep learningnetwork for implementing autonomous vehicle control. For example, deeplearning network 605 may be an artificial neural network such as aconvolutional neural network (CNN) that is trained using sensor data andused to output vehicle control results to vehicle control module 609. Invarious embodiments, deep learning network 605 is a multi-stage learningnetwork and can receive input data at two or more different stages ofthe network. For example, deep learning network 605 may receive featureand/or edge data at a first layer of deep learning network 605 andglobal data at a later layer (e.g., a second or third, etc. layer) ofdeep learning network 605. In various embodiments, deep learning network605 receives data at two or more different layers of the network and maycompress and/or downsize the data as it is processed through differentlayers. For example, the data size at layer one is a resolution that ishigher than the data at a subsequent stage. In some embodiments, thedata size at layer one is the full resolution of the captured image dataand the data at a subsequent layer is a lower resolution (e.g., aquarter of the size) of the captured image data. In various embodiments,the input data received from image pre-processor 603 at subsequentlayer(s) of deep learning network 605 matches the internal dataresolution(s) of the data that is processed through the one or moreprevious layers.

In some embodiments, artificial intelligence (AI) processor 607 is ahardware processor for running deep learning network 605. In someembodiments, AI processor 607 is a specialized AI processor forperforming inference using a convolutional neural network (CNN) onsensor data. In some embodiments, AI processor 607 is optimized for thebit depth of the sensor data. In some embodiments, AI processor 607 isoptimized for deep learning operations such as neural network operationsincluding convolution, dot-product, vector, and/or matrix operations,among others. In some embodiments, AI processor 607 is implemented usinga graphics processing unit (GPU). In various embodiments, AI processor607 is coupled to memory that is configured to provide the AI processorwith instructions which when executed cause the AI processor to performdeep learning analysis on the received input sensor data and todetermine a machine learning result used to at least in partautonomously operate a vehicle.

In some embodiments, vehicle control module 609 is utilized to processthe output of artificial intelligence (AI) processor 607 and totranslate the output into a vehicle control operation. In someembodiments, vehicle control module 609 is utilized to control thevehicle for autonomous driving. In some embodiments, vehicle controlmodule 609 can adjust the speed and/or steering of the vehicle. Forexample, vehicle control module 609 may be used to control a vehicle bybraking, steering, changing lanes, accelerating and merging into anotherlane, etc. In some embodiments, vehicle control module 609 is used tocontrol vehicle lighting such as brake lights, turns signals,headlights, etc. In some embodiments, vehicle control module 609 is usedto control vehicle audio conditions such as the vehicle's sound system,playing audio alerts, enabling a microphone, enabling the horn, etc. Insome embodiments, vehicle control module 609 is used to controlnotification systems including warning systems to inform the driverand/or passengers of driving events such as a potential collision or theapproach of an intended destination. In some embodiments, vehiclecontrol module 609 is used to adjust sensors such as sensors 601 of avehicle. For example, vehicle control module 609 may be used to changeparameters of one or more sensors such as modifying the orientation,changing the output resolution and/or format type, increasing ordecreasing the capture rate, adjusting the captured dynamic range,adjusting the focus of a camera, enabling and/or disabling a sensor,etc. In some embodiments, vehicle control module 609 may be used tochange parameters of image pre-processor 603 such as modifying thefrequency range of filters, adjusting feature and/or edge detectionparameters, adjusting channels and bit depth, etc. In variousembodiments, vehicle control module 609 is used to implementself-driving and/or driver-assisted control of a vehicle.

In some embodiments, network interface 611 is a communication interfacefor sending and/or receiving data including voice data. In variousembodiments, a network interface 611 includes a cellular or wirelessinterface for interfacing with remote servers, to connect and make voicecalls, to send and/or receive text messages, etc. For example, networkinterface 611 may be used to receive an update for the instructionsand/or operating parameters for sensors 601, image pre-processor 603,deep learning network 605, AI processor 607, and/or vehicle controlmodule 609. For example, a machine learning model of deep learningnetwork 605 may be updated using network interface 611. As anotherexample, network interface 611 may be used to update firmware of sensors601 and/or operating parameters of image pre-processor 603 such as imageprocessing parameters. In some embodiments, network interface 611 isused to make emergency contact with emergency services in the event ofan accident or near-accident. For example, in the event of a collision,network interface 611 may be used to contact emergency services for helpand may inform the emergency services of the location of the vehicle andcollision details. In various embodiments, network interface 611 is usedto implement autonomous driving features such as accessing calendarinformation to retrieve and/or update a destination location and/orexpected arrival time.

Although the foregoing embodiments have been described in some detailfor purposes of clarity of understanding, the invention is not limitedto the details provided. There are many alternative ways of implementingthe invention. The disclosed embodiments are illustrative and notrestrictive.

What is claimed is:
 1. A method, comprising: receiving an image capturedusing a sensor on a vehicle; extracting a global data component and afeature data component from the image which form input data, wherein theglobal data component is associated with global illumination data andthe feature data component is associated with edge data; providing theinput data to a convolutional neural network comprising a plurality oflayers, wherein the plurality of layers are sequential and formrespective portions of the convolutional neural network, wherein thefeature data component is provided as input to a first layer of theplurality of layers, wherein the global data component and anintermediate result output from a prior layer are provided as input to asecond layer of the plurality of layers, and wherein the second layer issubsequent to the first layer; and obtaining, based on a result of theconvolutional neural network, information indicating a vehicle controlresult which informs autonomous operation of the vehicle.
 2. The methodof claim 1, wherein subsequent to extraction, the global data componentis downsampled, and wherein the downsampled global data component isprovided as input to the second layer.
 3. The method of claim 1, whereina denoising filter is applied to the extracted global data component. 4.The method of claim 1, wherein the global data component is extractedvia a low-pass filter.
 5. The method of claim 1, wherein the featuredata component is extracted via a high-pass filter.
 6. The method ofclaim 1, wherein one or more of de-noising, demosaicing, local contrastenhancement, gain adjustment, and/or a thresholding process areperformed on at least a portion of the input data.
 7. The method ofclaim 1, wherein the first layer is an initial layer of theconvolutional neural network.
 8. The method of claim 1, wherein a thirddata component is extracted from the image via a band-pass filter, andwherein the third data component forms part of the input data.
 9. Themethod of claim 1, wherein the third data component is provided to athird layer of the convolutional neural network, and wherein the thirdlayer is subsequent to the first layer and prior to the second layer.10. The method of claim 1, wherein the vehicle control result isassociated with one or more of braking, steering, changing lanes,accelerating, and/or merging into a different lane.
 11. A computerprogram product, the computer program product being embodied in a isnon-transitory computer readable storage medium and comprising computerinstructions for: receiving an image captured using a sensor on avehicle; extracting a global data component and a feature data componentfrom the image which form input data, wherein the global data componentis associated with global illumination data and the feature datacomponent is associated with edge data; providing the input data to aconvolutional neural network comprising a plurality of layers, whereinthe plurality of layers are sequential and form respective portions ofthe convolutional neural network, wherein the feature data component isprovided as input to a first layer of the plurality of layers, whereinthe global data component and an intermediate result output from a priorlayer are provided as input to a second layer of the plurality oflayers, and wherein the second layer is subsequent to the first layer;and obtaining, based on a result of the convolutional neural network,information indicating a vehicle control result which informs autonomousoperation of the vehicle.
 12. The computer program product of claim 11,wherein subsequent to extraction, the global data component isdownsampled, and wherein the downsampled global data component isprovided as input to the second layer.
 13. The computer program productof claim 11, wherein the global data component is extracted via alow-pass filter.
 14. The computer program product of claim 11, whereinthe feature data component is extracted via a high-pass filter.
 15. Thecomputer program product of claim 11, wherein a third data component isextracted from the image via a band-pass filter, and wherein the thirddata component forms part of the input data.
 16. A system, comprising: aplurality of sensors on a vehicle; one or more processors and computerstorage media storing instructions that when executed by the processors,cause the processors to: receive at least one image from at least one ofthe sensors; extract a global data component and a feature datacomponent from the at least one image which form input data, wherein theglobal data component is associated with global illumination data andthe feature data component is associated with edge data; provide theinput data to a convolutional neural network comprising a plurality oflayers, wherein the plurality of layers are sequential and formrespective portions of the convolutional neural network, wherein thefeature data component is provided as input to a first layer of theplurality of layers, wherein the global data component and anintermediate result output from a prior layer are provided as input to asecond layer of the plurality of layers, and wherein the second layer issubsequent to the first layer; and obtain, based on a result of theconvolutional neural network, information indicating a vehicle controlresult which informs autonomous operation of the vehicle.
 17. The systemof claim 16, wherein subsequent to extraction, the global data componentis downsampled, and wherein the downsampled global data component isprovided as input to the second layer.
 18. The system of claim 16,wherein the global data component is extracted via a low-pass filter.19. The system of claim 16, wherein the feature data component isextracted via a high-pass filter.
 20. The system of claim 16, wherein athird data component is extracted from the image via a band-pass filter,and wherein the third data component forms part of the input data.